Skip to content

Optionally keep only inventory variables provided by inventory sources - #1202

Open
fernandorocagonzalez wants to merge 1 commit into
ctrliq:mainfrom
fernandorocagonzalez:feat-inv-prune-unmanaged-vars
Open

fernandorocagonzalez wants to merge 1 commit into
ctrliq:mainfrom
fernandorocagonzalez:feat-inv-prune-unmanaged-vars

Conversation

@fernandorocagonzalez

@fernandorocagonzalez fernandorocagonzalez commented Nov 13, 2026 •

Copy link
Copy Markdown
Contributor
SUMMARY

Since #1055, a sync with Overwrite variables removes the inventory variables that the same source wrote on its previous sync and no longer provides. That cleanup is careful on purpose, so some variables never go away:

  • variables written by syncs that ran before the source tracked what it writes (anything synced before 25.6.3),
  • variables added or edited by hand,
  • variables written by a source that was later deleted.

We hit the first case right after upgrading. A key renamed in group_vars/all kept its old value in the inventory, and the role kept preferring the old key, so the new one was ignored. Keys removed from the repo months ago are still sitting in the inventory variables, and the only way to clean them up is by hand, in every inventory that uses that source.

This adds an opt-in flag on the inventory, variables_from_sources_only. When it is set, every sync removes the inventory variables that no source of the inventory provides.

How it works:

  • New field Inventory.variables_from_sources_only, default false, so nothing changes for existing inventories. It's only accepted on regular inventories (it's rejected on smart ones and not exposed on the constructed and federated endpoints). It's copied along with the inventory.
  • In _update_inventory, when the flag is set, the sync keeps a variable only if the current source provides it or another source of the inventory recorded it on its own last sync (managed_inventory_variables from Remove stale inventory variables on sync when "Overwrite variables" is enabled #1055). Everything else is removed and logged at info level, so it shows up in the sync output. This doesn't depend on overwrite_vars.
  • A source that hasn't synced since the tracking was added has an empty record, and that looks the same as a source that synced and provides no variables. To tell them apart, there's a new non-editable InventorySource.managed_inventory_variables_recorded flag, set by every successful sync. While any other source of the inventory still has it unset, the sync removes nothing and logs a warning naming the sources that still need to sync. That way a sync of one source can't wipe the variables of a source that has never recorded them.
  • Everything happens under the same per-inventory perform_update advisory lock and transaction that Remove stale inventory variables on sync when "Overwrite variables" is enabled #1055 already uses, so syncs of different sources don't race, and a failed sync rolls back both the variables and the record.
  • Migration 0234.

UI: a Variables From Sources Only checkbox under Options in the inventory form, listed under Enabled Options in the details, with help text. Spanish translation added, and the other locales re-extracted.

Docs: docs/inventory/variables_from_sources_only.md, plus a mention in the inventories user guide.

ISSUE TYPE
  • New or Enhanced Feature
COMPONENT NAME
  • API
  • UI
  • Docs
ASCENDER VERSION
ascender: 25.6.4.dev37+ga4c3816b
ADDITIONAL INFORMATION

Tests added:

  • TestInventoryVariablesFromSourcesOnly in test_inventory_import.py. It covers: untracked and hand-added variables removed; kept when the flag is off; edited variables removed once no source provides them; works without overwrite_vars; variables of other sources kept; nothing removed until every source has synced; a source with no variables counts as synced; the variables of a deleted source removed on the next sync; a failed sync changes nothing.
  • API tests: the field is editable on regular inventories, rejected on smart ones, and absent on constructed ones. The copy test checks the field is preserved.
  • Vitest tests for the form checkbox and the detail view.

Checks from .github/copilot-instructions.md, run inside the dev container:

make test                 5436 passed, 6 skipped
make test_migrations      2 passed
check_migrations          No changes detected
check_settings            Every registered setting has a default its field accepts.
ruff format / ruff check / yamllint -s .   (tox linters pins)   clean
make ui-lint / ui-type-check               clean
make ui-test-screens      1979 passed
make ui-test-general      1180 passed
make ui-api-types         only the new field (3 schemas)

End to end on the docker-compose dev environment, with a git project whose inventory.yaml and group_vars/all/ feed an SCM source with overwrite variables on:

  1. Flag off: removing old_key from the repo and renaming renamed_from to renamed_to behaves as in Remove stale inventory variables on sync when "Overwrite variables" is enabled #1055. A key added by hand and a stand-in for a pre-upgrade leftover both stay.
  2. Flag on: the next sync removes both, and the sync output says so:
    INFO     Inventory variable "hand_added" removed, no source of this inventory provides it
    INFO     Inventory variable "legacy_cronat_allow" removed, no source of this inventory provides it
    
  3. A hand edit of a key the repo provides is overwritten by the next sync.
  4. Adding a second source that hasn't synced yet: a sync of the first one removes nothing and warns:
    WARNING  Inventory variables not provided by any source were kept: "src-second" must sync before they can be removed
    
    Once the second source syncs, the hand-added key goes away and the keys of both sources stay. Syncing the first source again keeps the second one's keys.
  5. Deleting the second source: the next sync of the first one removes the keys the deleted source wrote.
  6. Creating a smart inventory with the flag returns 400, the constructed inventory endpoint doesn't expose it, and copying an inventory keeps it.
  7. In the browser: the checkbox is saved, shows under Enabled Options, and the help text renders.

…sources

Overwrite variables only removes the inventory variables a source wrote
on its previous sync. Variables written before that tracking existed,
added or edited by hand, or written by a source that was later deleted
stay in the inventory forever, so a variable renamed in group_vars/all
keeps its old value and roles keep reading it.

The new variables_from_sources_only flag on regular inventories makes
every sync remove the inventory variables that no source of the
inventory provides. Since a source that has not synced since the
tracking was added has no record of its variables, nothing is removed
until every source has recorded them, tracked by the new
managed_inventory_variables_recorded field on the inventory source.

Signed-off-by: fernandorocagonzalez <fernandorocagonzalez@gmail.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Development

Successfully merging this pull request may close these issues.

1 participant